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Summary rel 2009-No. 071 

Analyzing performance by grade 10 
Hispanic high school students on the 
Massachusetts state assessment 



The report examines Hispanic high school 
students' performance on the Massa- 
chusetts Comprehensive Assessment 
System tests in English language arts and 
mathematics over 2002/03-2005/06. It 
compares the scores of grade 10 Hispanic 
and non-Hispanic students and uses 
multilevel regressions to examine asso- 
ciations between the Hispanic students' 
scores and student- and school-level 
characteristics. 

Massachusetts policymakers have been con- 
cerned about the consistently lower scores of 
Hispanic students compared with students in 
other subgroups on the state assessment — the 
Massachusetts Comprehensive Assessment 
System (MCAS) tests. To better understand 
Hispanic student characteristics and achieve- 
ment patterns, the Massachusetts Department 
of Elementary and Secondary Education asked 
the Regional Educational Laboratory North- 
east and Islands to analyze the performance 
of Hispanic students on the MCAS tests in 
English language arts and mathematics in 
high school. 

Two research questions drove this study: 

• How did the performance of grade 10 

Hispanic students on the MCAS English 
language arts and mathematics tests over 



2002/03-2005/06 compare with that of 
grade 10 non-Hispanic students? 

• Among grade 10 Hispanic students, which 
student- and school-level characteristics 
were associated with performance on the 
MCAS English language arts and math- 
ematics tests over 2002/03-2005/06? 

Descriptive analyses and t-tests were con- 
ducted to examine MCAS test performance for 
grade 10 Hispanic and non-Hispanic students 
in Massachusetts. Multilevel regression mod- 
eling was then used to analyze associations 
between Hispanic student MCAS achievement 
and student- and school-level characteristics. 

A different cohort of grade 10 Hispanic stu- 
dents was assessed for each school year. 

For the multilevel regressions the Office of 
Strategic Planning, Research, and Evaluation 
of the Massachusetts Department of Elemen- 
tary and Secondary Education provided 
student-level MCAS test performance data and 
background data for all grade 10 Hispanic stu- 
dents in Massachusetts over 2002/03-2005/06. 
Publicly available school-level performance 
and background data on all high schools in 
Massachusetts for the same period were ac- 
cessed through the Massachusetts Department 
of Elementary and Secondary Education web 
site (proffies.doe.mass.edu) and the Common 
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Core of Data of the U.S. Department of Educa- 
tion, National Center for Education Statistics 
(2006). 

In each school year from 2002/03 through 
2005/06 grade 10 Hispanic students scored 
significantly lower on the MCAS English 
language arts and mathematics tests than did 
grade 10 non-Hispanic students. However, the 
average scores for grade 10 Hispanic students 
in Massachusetts did increase over time hy a 
statistically significant amount in both content 
areas — a trend that has helped narrow this 
performance gap. 

The data for grade 10 Hispanic students were 
analyzed using multilevel regressions to deter- 
mine which student- and school-level variables 
showed a statistically significant relationship 
with student performance on the MCAS test 
over 2002/03-2005/06. For each school year 
statistically significant associations were found 
between several student-level variables and 
MCAS test scores: 

• Female Hispanic students scored signifi- 
cantly higher on the English language arts 
test than did male Hispanic students. Male 
Hispanic students scored significantly 
higher on the mathematics test than did 
female Hispanic students. 

• Hispanic students who were from low- 
income households, in special education, 
or limited English proficient or formerly 
limited English proficient — categorized 
as English proficient in the previous two 
years — scored significantly lower on both 
the English language arts and mathemat- 
ics tests than did students without those 
characteristics. (These associations mirror 



those typically reported in research on 
academic achievement for all racial/ethnic 
groups.) 

• Hispanic students from Caribbean coun- 
tries, Central American countries, and 
Mexico scored significantly lower on the 
English language arts test than did U.S.- 
born Hispanic students. Hispanic students 
from South American countries other 
than Brazil scored significantly higher on 
the mathematics test than did U.S.-born 
Hispanic students. 

For each year from 2002/03 through 2005/06 a 
statistically significant association was found 
between MCAS test performance and only one 
school-level variable: 

• Hispanic students in schools with higher 
attendance rates scored significantly 
higher on both the English language arts 
and mathematics tests than did Hispanic 
students in schools with lower attendance 
rates, all other variables held constant. 

The study has several limitations, four of 
them especially important. The multilevel 
regressions describe statistical associations 
rather than causal relationships between 
student- and school-level characteristics 
and Hispanic students’ MCAS test scores. 

The large share of data excluded from the 
analyses — most of it missing data — might 
have biased the findings, since excluded 
students appear to have lower achievement 
than included students. Many variables that 
could help to explain differences in academic 
achievement patterns for Hispanic students 
were not analyzed. And the analyses do not 
account for possible associations between 
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Hispanic students’ test scores from 2002/03 
through 2005/06 and a change in federal test- 
ing policy in February 2004. 

To better understand the academic achieve- 
ment of Hispanic high school students, 
further research is suggested. Such re- 
search should examine additional statistical 



relationships — both among various demo- 
graphic and student- and school-level char- 
acteristics (some of which this study did not 
consider) and between certain characteristics 
and the MCAS test scores of Hispanic and 
non-Hispanic students. 

June 2009 
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The report examines 
Hispanic high 
schooi students' 
performance on 
the Massachusetts 
Comprehensive 
Assessment System 
tests in English 
language arts and 
mathematics over 
2002/03-2005/06. 

It compares the 
scores of grade 10 
Hispanic and non- 
Hispanic students 
and uses multilevel 
regressions 
to examine 
associations 
between the 
Hispanic students' 
scores and student- 
and school-level 
characteristics. 



WHY THIS STUDY? 

Massachusetts policymakers recently expressed 
a desire to better understand Hispanic student 
achievement patterns in their state. Scores on the 
Massachusetts Comprehensive Assessment System 
(MCAS) tests have consistently revealed a gap 
in performance between Hispanic students and 
students from other subgroups, a gap correspond- 
ing to national trends. Yet studies that look closely 
at achievement in the national Hispanic student 
population are rare, and no previous analyses 
of Massachusetts data have been done for this 
subgroup. 

To reveal achievement patterns for the diverse 
group of Hispanic students in Massachusetts 
and to inform policy and program decisions, the 
Massachusetts Deputy Commissioner of Educa- 
tion and staff at the Massachusetts Department of 
Elementary and Secondary Education asked the 
Regional Educational Laboratory Northeast and 
Islands to analyze the scores of grade 10 Hispanic 
students on MCAS tests in English language arts 
and mathematics over 2002/03-2005/06. The 
department wished to learn about associations 
between the scores and school- and student-level 
characteristics, such as school attendance rate, 
English proficiency status, country of origin, first 
language, and school attended. (See appendix A 
on the characteristics identified in research on 
Hispanic student academic achievement.) 

The report compares the MCAS test performance 
of grade 10 Hispanic students with that of grade 
10 non-Hispanic students over 2002/03-2005/06, 
looking at both English language arts and mathe- 
matics. Multilevel regressions are used to examine 
relationships between Hispanic students’ MCAS 
test performance and student- and school-level 
characteristics. (Key terms, including Hispanic 
student, are defined in box 1.) 

Two research questions drove the study: 

• How did the performance of grade 10 His- 
panic students on the MCAS English language 
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BOX 1 

Definitions of key terms 

Limited English proficient student. A 
student with a first or native lan- 
guage other than English and who 
is incapable of performing ordinary 
classwork in English. 

Hispanic student. A student who 
identifies his or her culture or origin 
as Central American, Cuban, Mexi- 
can, Puerto Rican, South American, 
or other Spanish culture or origin. 

A Hispanic student can come from a 
non-Spanish-speaking country. ('This 
definition, at www.doe.mass.edu/in- 
foservices/data/guides/race_faq.html, 
is identical to that used by the U.S. 
Census Bureau, at www.census.gov/ 
population/www/socdemo/hispanic/ 
hispdef html. Differences in report- 
ing data on Hispanic students for 
2006 and prior years are discussed in 
appendix A. For information on how 
the Massachusetts Department of 
Elementary and Secondary Education 
categorizes Hispanic students, see 
appendix B.) 

Low income. Low income is de- 
fined as meeting any one of three 
criteria — eligible for free or reduced- 
price lunch, receiving Transitional 
Aid to Needy Families benefits, or 
eligible for food stamps. The infor- 
mation is collected from students 
by the Massachusetts Department 
of Elementary and Secondary 
Education. 

Multilevel regression modeling. A 
set of regression-based procedures 
used to analyze data with a nested 
or hierarchical structure (such as 
students nested within schools). 



Multilevel regression modeling ac- 
counts for correlated errors among 
individuals, allows the relation- 
ship between independent and 
dependent variables to vary across 
groups, and allows individual and 
group characteristics to be included 
in predictive models of individual 
outcomes. 

Non-Hispanic student. Any student 
not self-identified as Hispanic. 

Performance level. The degree to 
which a student shows mastery of 
state standards as measured by the 
Massachusetts Comprehensive As- 
sessment System (MCAS) tests. The 
four levels, based on scale scores, are 
warning, needs improvement, profi- 
cient, and advanced. The minimum 
scale score needed for each profi- 
ciency level varies by year (see table 
B6 in appendix B). 

Raw score. A student’s total score 
across all items for each MCAS test, 
without scaling. Possible raw score 
ranges vary by test and year. (For an 
explanation of how raw scores were 
used, see appendix B.) 

Scale score. An MCAS raw score 
converted to a common scale through 
a standard-setting process, MCAS 
scale scores are the minimum scores 
for partial, solid, and sophisticated 
understanding of curriculum frame- 
works for any grade content stan- 
dard (Massachusetts Department of 
Elementary and Secondary Education 
2003). The scale for grade 10 MCAS 
scores ranges from 200 to 280. (See 
tables B4 and B5 in appendix B for 
conversion charts for raw to scale 
scores.) 



School. A Massachusetts school 
with publicly available school-level 
data on the Massachusetts Depart- 
ment of Elementary and Second- 
ary Education web site. (The web 
site did not have school-level data 
for designated special education 
schools so they are excluded from 
the samples.) 

School-level characteristics. These 
variables — such as dropout rate and 
percentage of students from low- 
income families — are defined in 
appendix B. 

Standard deviation. This is a mea- 
sure of how widely or narrowly data 
are dispersed around the mean for 
the distribution. A student’s test 
score can be described in terms of 
standard deviation units by sub- 
tracting the mean from the student’s 
score and dividing that figure by the 
standard deviation. 

Standard error. This is a measure of 
the amount of error between a statis- 
tic estimated from a sample and the 
true value for the population. 

Student. A student is a grade 10 
Massachusetts student for whom 
two types of data are available: 
MCAS test performance data for 
English language arts and math- 
ematics and corresponding school- 
level data. 

Student-level characteristics. These 
variables — for example, first lan- 
guage and low-income status — are 
defined in appendix B. 

Variance. Variance is the squared 
standard deviation. 
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arts and mathematics tests over 2002/03- 
2005/06 compare with that of grade 10 non- 
Hispanic students? 



improved significantly, by 4 points (from 242 to 
246), but their scale scores then remained constant 
until 2005/06. 



• Among grade 10 Hispanic students, which 
student- and school-level characteristics were 
associated with performance on the MCAS 
English language arts and mathematics tests 
over 2002/03-2005/06? 

The study methods are summarized in box 2. 

A large percentage of data were removed before 
the final analyses. (The datasets are described 
in appendix B. The data removal procedures are 
described in appendix C.) 



HOW DID PERFORMANCE BY GRADE 10 HISPANIC 
STUDENTS ON THE STATE ASSESSMENT OVER 
2002/03-2005/06 COMPARE WITH PERFORMANCE 
BY GRADE 10 NON-HISPANIC STUDENTS? 

To answer the first research question, scale scores 
(based on mean raw scores) for Hispanic and 
non-Hispanic students were examined to find 
variations in the performance gap between them 
over 2002/03-2005/06 (see box 3 for a summary 
of characteristics of Hispanic and non-Hispanic 
students). Then t-tests were used to determine 
whether average raw scores for Hispanic and 
non-Hispanic students showed statistically 
significant differences and whether scores for 
Hispanic students improved significantly over 
time. 



Differences in performance on the state assessment 
by grade 10 Hispanic and non-Hispanic students 

In each study year the average scale score of His- 
panic students on the MCAS English language arts 
test was lower than that of non-Hispanic students 
by a statistically significant amount (figure 1 and 
table 1). However, the average scale scores in Eng- 
lish language arts for Hispanic students improved 
significantly between 2002/03 and 2005/06, by 12 
points (from 222 to 234). Between 2002/03 and 
2004/05 non-Hispanic students’ scale scores also 



Similarly, in each study year the average scale 
score for Hispanic students on the MCAS math- 
ematics test was lower than that for non-Hispanic 
students by a statistically significant amount 
(figure 2 and table 2). However, between 2002/03 
and 2005/06 the average scale scores in mathemat- 
ics for both Hispanic and non-Hispanic students 
increased significantly. The average scale score for 
Hispanic students rose 10 points, while that for 
non-Hispanic students rose 8 points. 

The changes in MCAS test performance gaps 
between Hispanic and non-Hispanic students 
are shown in figure 3. The gap for English lan- 
guage arts was 20 points in 2002/03, 22 points in 
2003/04, 20 points in 2004/05, and 12 points in 
2005/06. The gap for mathematics was 20 points 
in 2002/03, 2003/04, and 2004/05, and 18 points 
in 2005/06. The declines in both performance gaps 
between 2003/04 and 2005/06 are attributable to 



FIGURE 1 

Average scale scores on the Massachusetts 
Comprehensive Assessment System English 
language arts for grade 10 Hispanic and non- 
Hispanic students in Massachusetts, 2002/03- 
2005/06 



Average scale score 

280 ■ Hispanic students 

■ Non-Hispanic students 

260 



246 246 246 




2002/03 2003/04 2004/05 2005/06 



Source: Authors' analysis based on data from Massachusetts Elementary 
and Secondary Education Department, Office of Strategic Planning, 
Research and Evaluation 2007. 
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ANALYZING PERFORMANCE BY HISPANIC HIGH SCHOOL STUDENTS ON THE MASSACHUSETTS STATE ASSESSMENT 



BOX 2 

Methods 

The study used two types of data: 

• Background and MCAS perfor- 
mance data on grade 10 Hispanic 
students in Massachusetts for 
2002/03-2005/06, provided by 
the Massachusetts Department 
of Elementary and Secondary 
Education, Office of Strategic 
Planning, Research, and Evalua- 
tion (2007). 

• Publicly available data on all 
high schools in Massachusetts 
for 2002/03-2005/06, accessed 
through the Massachusetts 
Department of Elementary and 
Secondary Education web site 
(Massachusetts Department of 
Elementary and Secondary Edu- 
cation 2007b) and the U.S. De- 
partment of Education, National 
Center for Education Statistics 
(2006) Common Core of Data. 



(Special education schools and 
their students were not included 
in the analyses.) 

A large share of data was removed 
from the study, mainly because of 
missing student- or school-level data 
(see appendix C for details). Compar- 
isons of included and removed cases 
for each student- and school-level 
variable revealed that the removal 
of so much data might have biased 
the findings, since removed students 
appeared to have lower achievement 
than included students. 

Descriptive analyses compared the 
performance of grade 10 Hispanic stu- 
dents on the English language arts and 
mathematics MCAS tests with that of 
grade 10 non-Hispanic students. T- 
tests were used to determine whether 
the two groups’ scores differed signifi- 
cantly on each subject in each school 
year and whether significant improve- 
ments occurred over time. To examine 
associations between student- and 



school-level characteristics and 
Hispanic student MCAS performance, 
multilevel regression modeling was 
used for the population of students 
self-reporting as Hispanic. 

This study did not follow any student 
cohort over time. Therefore, it makes 
no claims about whether differences 
in cohort characteristics are associ- 
ated with student performance over 
time. In the multilevel regression 
modeling, the regression coefficient 
for a given variable in the model as- 
sumes that all other variables are held 
constant — so although previous re- 
search might have found an associa- 
tion between the given variable and 
student test performance, this study 
might not find that association to be 
significant after accounting for other 
variables (see appendix A for review 
of previous literature). Results of the 
multilevel regressions are provided 
as standard deviation differences for 
each variable and as absolute changes 
in raw score points for each variable. 



yearly increases in Hispanic student scale scores 
(see figures 1 and 2) — with the increases being 
greater for the English language arts test (see 
figure 1). 



AMONG GRADE 10 HISPANIC STUDENTS, 

WHICH STUDENT- AND SCHOOL-LEVEL 
CHARACTERISTICS WERE ASSOCIATED 
WITH PERFORMANCE ON THE STATE 
ASSESSMENT OVER 2002/03-2005/06? 

To answer the second research question, multilevel 
regression modeling was used to analyze statistical 
relationships between student- and school-level 
variables and Hispanic students’ MCAS test scores. 
Such relationships are termed statistically signifi- 
cant when the regression coefficient associated 



with a variable is statistically significantly differ- 
ent from zero. 



Overview of findings 

for each year from 2002/03 through 2005/06 
multilevel regression modeling revealed statisti- 
cally significant associations between several 
student-level variables and MCAS test scores 
(see box 3 for a summary of student-level 
characteristics): 

• Female Hispanic students scored significantly 
higher on the English language arts test than 
did male Hispanic students. Male Hispanic 
students scored significantly higher on the 
mathematics test than did female Hispanic 
students. 



WHICH CHARACTERISTICS WERE ASSOCIATED WITH PERFORMANCE OF HISPANIC STUDENTS ON THE ASSESSMENT? 
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BOX 3 

Characteristics of grade 
10 Hispanic and grade 10 
non-Hispanic students in 
Massachusetts in 2005/06 

In 2005/06 there were 7,394 grade 
10 Hispanic students in Massachu- 
setts (see table), or 10.6 percent of 
the state’s 10th graders, up from 9.0 
percent in 2002/03. (Characteristics 
of grade 10 Hispanic and non- 
Hispanic students in Massachusetts 



for 2002/03-2005/06 are in appen- 
dix D.) 

There were several similarities be- 
tween the Hispanic and non-Hispanic 
students in 2005/06: 

• Each group was about evenly 
split between male and female 
students. 

• Special education students made 
up similar shares of the Hispanic 



(16.8 percent) and non-Hispanic 
(14.0 percent) student populations. 

• More than 90 percent of students 
in each group were born in the 
United States. 

There were also notable differences in 
characteristics between Hispanic and 
non-Hispanic students: 

• Among Hispanic students, 67.6 
percent were from low-income 

(CONTINUED) 



Characteristics of grade 10 Hispanic and non-Hispanic students in Massachusetts, 2005/06 





Hispanic students (N 


= 7,394) 


Non-Hispanic students (N = 62,139) 


Student characteristic 


Percent 


Number 


Percent 


Number 


Gender 


Female 


50.6 


3,738 


49.8 


30,973 


Male 


49.4 


3,656 


50.2 


31,166 


Socioeconomic status 


Low income 


67.6 


4,997 


18.3 


11,341 


Not low income 


32.4 


2,397 


81.7 


50,798 


Special education status 


Special education 


16.8 


1,244 


14.0 


8,681 


Not special education 


83.2 


6,150 


86.0 


53,458 


English proficiency status 


English proficient 


78.3 


5,787 


97.9 


60,571 


Limited English proficient 


8.0 


1,015 


1.5 


928 


Former limited English proficient® 


8.0 


592 


1.0 


640 


First language 


English 


40.7 


3,008 


92.5 


57,497 


Portuguese 


2.9 


218 


1.1 


689 


Spanish 


55.4 


4,009 


0.2 


119 


Other languages 


0.9 


69 


6.2 


3,834 


Country of origin 


Brazil 


0.5 


40 


0.1 


70 


Caribbean country 


2.2 


164 


0.2 


126 


Central American country or Mexico 


2.2 


166 


0.0 


5 


South American country other than Brazil 


0.9 


70 


0.0 


2 


United States 


93.9 


6,940 


98.5 


61,197 


Other countries 


0.2 


14 


1.2 


739 



a. Students newly categorized as English proficient during the previous two years. 

Source.- Authors' analysis based on data from Massachusetts Elementary and Secondary Education Department, Office of Strategic Planning, Research, and 
Evaluation (2007). 




